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Abstract — In this paper, we study the use of Gr(q)-quantized 
LDGM codes for binary source coding. By employing quantiza- 
tion, it is possible to obtain binary codewords with a non-uniform 
distribution. The obtained statistics is hence suitable for optimal, 
direct quantization of non-uniform Bernoulli sources. We employ 
a message-passing algorithm combined with a decimation proce- 
dure in order to perform compression. The experimental results 
based on GF(q)-LDGM codes with regular degree distributions 
yield performances quite close to the theoretical rate-distortion 
bounds. 

I. Introduction 

In the proof of achievability of the channel capacity, as well 
as in the proof of achievability of the rate-distortion function, 
random ensembles of codes are tested and shown to be asymp- 
totically optimal 1 1]. But, in practice, random codes are useless 
because without any structure no channel decoding/source 
encoding algorithms exist that run in reasonable time. In 
many cases of interest (e.g. binary symmetric channels, binary 
source coding with Hamming-distance distortion measure, 
...) optimality is also achieved by linear codes. However, 
conducting a search among their codewords is still a tough 
task unless some other condition holds. 

Luckily, sparse structures, e.g. linear codes with parity- 
check or generator matrices with a number of non-zero el- 
ements that is roughly proportional to the length of their 
codewords, have been shown to asymptotically perform very 
close to the theoretical bounds. Notably, for these structures, 
there often exist practical algorithms that essentially permit to 
reach their asymptotic performances. 

For example, in the case of channel coding for the binary 
symmetric channel, the low density parity-check (LDPC) codes 
are shown to almost achieve the channel capacity IS), at least 
under maximum-likelihood decoding. In addition, they can be 
decoded in linear time with respect to the codeword length 
by using the standard message-passing algorithm [3] with 
negligible performance losses [4|. 

On the other side, it has been proved that sparse codes 
are optimal too for lossy source coding. For example, low 
density generator matrix codes (LDGM) almost achieve the 
rate-distortion function of the binary-symmetric memoryless 
source with Hamming-distance distortion measure Q; also, 
by appropriately combining LDPC and LDGM codes it is 
possible to solve problems of coding with side information |6|. 
But, if the standard message-passing algorithm Q is employed 
in this dual setting, where there actually exist multiple near- 



optimal encodings, it usually fails to converge or converges to 
meaningless marginals. 

One of the first works showing that iterative algorithms 
can actually work as well for quantization appeared in |7l|, 
where the problem of binary erasure quantization over LDGM 
codes was tackled. In order to solve the convergence problem, 
a simple modification of the message-passing algorithm was 
proposed in |8| that led to satisfying results. Other works in- 
vestigate adaptations of algorithms from the field of statistical 
physics in order to do data compression as well, e.g. by using 
the survey propagation algorithm ||9|. Furthermore, practical 
iterative algorithms that rely on non-linear codes have been 
proposed for achieving the rate-distortion bound \ 

One interesting possible workaround to achieve convergence 
in binary source coding with LDGM codes, as suggested in 
is), is to (i) run the traditional message-passing algorithm over 
an augmented code-space where variables can also take a/ree 
state, (ii) set the more certain variables to their preferred (non- 
free) states, and (iii) iterate this procedure over the decimated 
code obtained by eliminating the variables that have been set in 
step (ii), until all variables have been set. Indeed, this approach 
can be seen as extending both the standard message-passing 
rules and the survey propagation ones ifTTI : in addition, it can 
be simplified as shown in [12J. 

In this paper we aim at extending the algorithms described 
in ||5l to the case in which the binary source to be compressed 
is not uniform. In this case, in fact, a linear binary code 
whose codewords have a uniform statistics is no longer optimal 
111, as happens in the dual case of channel coding for non- 
symmetric channels. In the latter case optimality can still be 
achieved relying on non-binary LDPC codes and their variants, 
such as for example the GF{q)-quantized coset LDPC (GQC- 
LDPC) codes |13|. Among others, GQC-LDPC codes have 
interesting applications in the framework of channel coding 
with side information at the encoder, where they are applied 
in conjunction with good source codes in a superposed fashion 
fl4l . In particular, in this paper, we define GF{q)- quantized 
LDGM (GQ-LDGM) codes as duals of GQC-LDPC codes, 
and design practical algorithms for quantization over their 
codewords. GQ-LDGM codes can be eventually employed in 
conjunction with good channel codes for source coding with 
side information at the decoder, using a superposition scheme 
dual to the one in |14|. 

As the question "can linear codes with a suitable mapping 
function achieve the rate-distortion bound?" was positively 



answered in ifTSl . and similar results have been proved for 
LDGM codes fTSl, our focus on the practical side of the 
problem, namely on designing practical encoding algorithms, 
is fully justified. Other than this, constructions based on quan- 
tized LDGM codes (or on multilevel quantization) were used 
as well to tackle scalable coding [17,1 and multiple description 
coding ifTSl problems. This paper is also connected with ifTOll . 
where codes for non-uniform sources too are designed and 
tested. 

The rest of this paper is organized as follows. In Section Ull 
we review the definition of GF((7)-LDGM codes and extend 
the space-augmentation procedure used in |5| to these codes. 
In Section |lll] we introduce GQ-LDGM codes and describe 
the message-passing and decimation rules used for performing 
quantization. In Section |IV] we present some experimental 
results obtained with regular degree GQ-LDGM codes. Section 
rvl summarizes our conclusions. 




LDGM code 

Fig. 1. Factor graph of an LDGM code completed with the compatibihty 
functions for assigning suitable probabilities to its generalized codewords. 



II. GF((7)-LDGM Codes and Their Augmentation 

An (n,m) LDGM code over GF{q) = {Iq = 
0,li, . . . ,lq-i} is easily specified by means of a factor 
graph 121 made by n checks a and n constrained vari- 
ables Xa, a £ {0,1,..., n — 1}, plus m free variables Zi, 
i e {0,1,..., m — 1}. Each check a is connected to Xa 
on one side and to z^rja) on the othefl where N{a) C 
{0,1,..., m — 1}; each such connection is labelled with a 
weight gia G GF{q)\{lo}- Each variable Zi is then connected 
to the checks in N{i) C {0, 1, . . . , n~l}. Variables take values 
on GF{q), checks are satisfied iff Xa — J2i£N(a) 9iaZi, and 
the [n + to) -tuples (x, z) such that all checks are satisfied 
form the codewords of the LDGM code. 

The augmented LDGM code is made of all the (n + m)- 
tuples of elements of GF{q)* = GF{q) U {*} such that, with 
reference to the factor-graph of the original code (see Fig. [T), 

1) either checks connect to Xa = * on one side and at least 
one element in i.isiia) equals * on the other, or they are 
satisfied (i.e. their neighborhood is made of all elements 
of GF{q) that satisfy the condition given above); 

2) free variables different from * connect to at least two 
exactly satisfied checks (we assume that |A^(j)| > 2). 

Those codewords are called generalized codewords. 

Due to the constraint in|2] in order to check if an (n + m)- 
tuple is a generalized codeword, not only additional compat- 
ibility functions must be imposed on the free variables, but 
also the corresponding nodes should be equipped with some 
knowledge about the state of the checks to which they connect. 
Similarly to what done in |5|, this can be easily achieved 
by replacing the nodes corresponding to the free variables 
with some variables P, e "P,; = GF{q) x {1, 2, . . . , 2l^(*)l - 
1} U {*}, where {1, 2, . . . , 2l^(*)l - 1} is assumed in one- 
to-one correspondence with the non-empty sets in V {N{i)), 
the power set of N{i). Once we define the two projections 
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A C N{i), if = (•, index of subset A) ' 
and the compatibility functions at checks and node^ (|3]l and 

^[{P^)^X[P^ = *yHP^)\>^] , (2) 

the generalized codewords are all the (n+TO)-tuples (x, 7r(P)) 
such that all Xa and Pi satisfy the new compatibility functions. 
In practice. Pi brings both the value of the free variable, and, 
in case of the value being different from the indices of the 
checks that are enforcing this value (note that at least one such 
check always exists). 

III. Binary Source Compression with 

GF(q')-QUANTIZED LDGM CODES 

At this point, when one wants to use the original LDGM 
code in a quantized fashion for quantizing a random realization 
of a binary codeword, a map Q : GF{q)* GF(2)* is intro- 
duced such that Q{xa) = * iff Xa = *. This map eventually 
converts a GF{q) codeword corresponding (through standard 
LDGM encoding) to a compressed representation in GF{q) 
into the GF{2) codeword used as source reconstruction. 

Then, by adding the compatibility functions 
for the constrained variables ij;a{xa; Sa, l3,Ws) = 
ipi^'' {Q{xa);sa,l3,Ws), where 
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if Xa — Sa 

if xi^) = * 

if Xa — Sa 



(3) 



and by using ipi[Pi]Wi) — wf^'^' *^^p'^{Pi) as compatibility 
functions for the free variables, a probability distribution over 



'zjv(a) Stands for {zi : i g N{a)}. 



^x['] is one iff the condition indicated is satisfied and zero otherwise; it is 
also assumed that ifc + * = * + 'fe = * and * + * = *. 
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all (n + TO)-tuples (x, P) is enforced. This distribution, when 
marginalized over all the (x, P) tuples corresponding to the 
same generalized codeword (x, z), leads to a probability 



^*(^) . w'^'i^} . g-2/3d„(Q(x).s) 



P[(x, z);Sa,/3,Ws,'!«i] OC 

(4) 

The factor graph describing this distribution is shown in 
Fig. [T] In the formulas, s denotes the n-dimensional source 
realization to be quantized, /3 > determines how much more 
probability should be given to the values (taken by constrained 
variables) that after quantization match the source with respect 
to those that will not match it, and Ws > and Wi > 
determine how much each value * (occurring in constrained 
or free variables, respectively) modifies the overall generalized 
codeword probability. In particular, Ws > 1 (or Wj > 1) will 
make the * value to be preferred with respect to symbols in 
GF{q). Also, N*{-) denotes the number of * values in a given 
tuple of GF{q)*, and dni-, •)) denotes the Hamming distance 
between tuples of GF{2)*, assuming that * has distance 1/2 
from any other symbol. 

Despite the exponentially large cardinality of Vi, that in 
practice seems to prevent the utilization of message-passing 
algorithms, it can be shown that the messages over each edge 
can be described by only q + 2 probabilities, at least in the 
case where g is a prime number. In particular, in the check-to- 
variable direction these probabilities are /i^-, and 
for k = 0,1, . . . ,q — 1, because the messages satisfy 
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In the variable-to-check direction, it is similarly sufficient 
to specify the probabilities ^*^, fif^, and for k — 

0, 1, . . . , g — 1, which are defined as 



(8) 
(9) 



(10) 

With these definitions, the messages from check a to 
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variable i can be computed by 
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The messages from variable i to check a can be instead 
computed by 
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q-1 

E n 

k=0 ceN(i)\{a} 



(14) 



^ n 

ceW(i)\{a} 



E 

h^N(i)\{a] 



E^i' 



./£ = 



(15) 



Mi, 



n 

ceA'('i)\{a,6} 

n ^^i^f'^- n ^tm 

ceN{i)\{a} ceN{i)\{a} 



Finally, the marginals with respect to each Zi are found 
using the following equations: 
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(18) 



where A; = 0, 1, . . . , g — 1. 

As a remark, the derivation of these message-passing rules 
is quite straightforward, despite of combinatorial nature. Due 

^['^j'a 'l denotes the column vector whose g components are obtained for 
k = 0, 1, . . . , q — 1; also, components of product of vectors are assumed 
to be products of the corresponding components. DFTq and IDFTq denote 
5-point Fourier transformation and its inverse, respectively. As a remark, we 
were able to derive the given formula for [A^'j'''] only for prime values of g. 



to the limited space, the actual derivations were not included; 
however, one can notice how these message-updating rules 
appear as an extension of the ones relative to the GF{2) field 
and derived in ||5l. 

In practice, the quantization algorithm we designed follows 
the following steps: 

1) All check-to-variable messages are initialized by as- 
suming that there are no incoming variable-to-check 
messages; in particular, this implies — Ws, /i^j — 0, 
and /xjj'/"^ = ipaigiah); 

2) free variables that are either disconnected from any 
check or that connect exclusively to degree one checks 
are removed from the factor-graph; 

3) message-passing rules for computing variable-to-check 
messages and recomputing check-to-variable messages 
are iteratively applied until convergence is achieved (or 
a maximum number of iterations is reached); 

4) free variables that are sufficiently biased are removed 
from the factor-graph; 

5) if there are still variables to be removed, go back to step 
[T] otherwise exit the algorithm. 

Each time a free variable Zi is removed, its value is set as 
the /fc in GF(q) that maximizes pf*''; edges connecting that 
variable to checks are removed as well, and the compatibility 
functions ifja, for each a e N{i), are updated in order to take 
into account for this removal. Of course, unconnected checks 
are removed too from the factor-graph. 

In addition, in order to quantize the bias of a probability 
distribution over GF{q)*, we used the unbiased standard 



deviation of the vector [p. 



Bi equals 



S, i.e. p* 



multiplied by y/g. In particular, 
1 — 5, it turns out that the bias 
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q-1 



(19) 



which is always in the range < i?i < 1. As a remark, note 
that the maximum is achieved iff S* = 1 (i.e. p* — 0) and 
exactly one element of [pf ''^] equals one. This formulation 
extends the bias concept introduced in ||5|. 

IV. Experimental Results 

In the experiments we defined the quantizer Q( ) on the 
base of a parameter < Qm < q, that specifies the minimum 
value in GF{q) to be quantized as the non-zero value of 
GF{2). In practice, Q{lk) = Iq for all k < Qm, while 
Q{lk) = h otherwise. Hence, the employed GQ-LDGM 
codes have elements assuming the value li with probability 
r = . We tested 4 different combinations of q and Qm, 

as shown in Table U 

In these preliminary experiments, we derived the GQ- 
LDGM codes from randomly generated GF((7)-LDGM codes 
with regular degree distributions. In particular, checks and 
free variables have degree dc and dy respectively, so that the 
resulting (source coding) rate, expressed in bit/sample units. 



TABLE I 

Degrees, rates and source probability used in the various 
experiment sets. 



Experiment set 


1 


2 


3 


4 


<? 


5 


3 


5 


2 


Qm 


4 


2 


3 


1 


r 


0.200 


0.333 


0.400 


0.500 


dc 


2 


2 


2 


2 


dv 


9 


6 


9 


4 


R [bit/sample] 


0.516 


0.528 


0.516 


0.500 


Ps 


0.230 


0.365 


0.420 


0.500 



equals R = log2((7). We fixed the check degree dc = 2 and 
derived, for each employed value of q, a suitable value of dy 
for obtaining a rate of approximately 0.5 bit/sample, as shown 
in Table |T0 

In order to use GQ-LDGM codeword statistics that match 
the optimal distribution, i.e. the one achieving the theoretical 
rate-distortion bound at the employed rates, in each experi- 
ment we generated a random i.i.d. source in GF{2) with a 
probability of symbol li equal to the values of ps shown in 
Table |T] These values are such that, approximately, 

D{R;Ps) 



Ps 



(20) 



l-2D{R;p,) 

where D{R;ps) is the theoretical (Hamming) distortion at the 
employed rates R, according to the Bernoulli distortion-rate 
function U. In particular, D satisfies H{D) — H{ps) — R, 
where H{-) is the entropy of the binary distribution with a 
given symbol probability. 

In all experiments we assumed that the message-passing 
procedure relative to the various decimation steps converged 
when the maximum absolute difference between the (normal- 
ized) marginals computed in the current iteration 
and the ones computed in the previous one is less than 
MPth = 0.05, for all free-variables. Also, we fixed the 
maximum number of iterations at A/Pmax = 100, which was 
typically reached in the first couple of decimation steps. 

Similarly to what done in |5|, after each message-passing 
run we removed at least Vm — 1% of the variables (starting 
from the more biased ones), as well as all the variables with 
a bias greater or equal than Bm = 0.7, until a maximum of 
rm = 10% of the variables was removed. We also borrowed 
from ||5| the choices of Wi — exp(0.05) and Ws = cxp(O.lO). 
Instead, in order to suitably choose (3 accordingly to the 
employed rate (and source statistics), we propose to choose 
it in a way such that the compatibility function ipaixa) 
corresponds to the likelihood function of the test channel IT] 
related to the source coding problem at hancjl; in practice, we 

''As noted in and (8), in the case of in'egular degree distributions it is 
important to have an high ratio of checks of degree 2. This is the reason why 
we chose dc = 2 in our experiments with regular distributions. Note that 
this choice, in case of g = 2, introduces linear dependency between the rows 
of the LDGM generating matrix. In practice, the actual rate could be (very 
slightly) reduced as there always exist 2 m-tuples of free-variables leading to 
the same LDGM codeword. 

^This is reasonable as we may assume that the source (with probability ps) 
is in fact a corrupted version of a codeword (with probability r), obtained after 
the latter went through the test channel (with probability of error D(R\ Ps)). 
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Fig. 2. Empirical performance at ij = 0.5 bit/sample compared with tiie 
rate-distortion and time-sharing bounds and with some data from 1101 . 
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(21) 



which equals approximately 1.53, 1.19, 1.10, and 1.05, re- 
spectively for each experiment set. 

The results, reported in Fig. |2] are relative to compression 
of random realizations of length n = 10^, 10"* and 10^; 
they have been averaged over 4 GQ-LDGM codes with 100 
compressions/code, 10 codes with 500 compressions/code, and 
10 codes with 1000 compressions/code, respectively for the 
three lengths. From the plot, where the rate-distortion bound 
H{D) = H{ps) — R and the time-sharing bound (between the 
zero-rate and the zero-distortion points) 

H{ps) - R 



D^Ps 



(22) 



H{ps) 

are given too, we concluded that the proposed algorithm 
performed quite close to the rate-distortion bound, and al- 
ways better than the time-sharing bound. Surprisingly, the 
performance turned out to be essentially independent from the 
codeword length. 

The algorithm was implemented in C and was run on an 
Intel Pentium 4 CPU operating at 3 GHz, with 1.5 GByte of 
RAM. In terms of execution speed it turned out to be very 
competitive with respect to the timings given in |5|. In fact, 
in the experiments of set 1, 80.53, 6.96 and 0.52 seconds were 
necessary on average for performing each source compression, 
respectively for realizations of length n — 10^, 10^ and 10"^. 
In the experiments of set 2-4 the timings are 53.27, 3.98, 
and 0.31; 78.70, 6.93, and 0.54; and 46.74, 2.28, and 0.18, 
respectively. 

V. Conclusion 

In this paper we presented a novel message- 
passing/decimation algorithm for binary source quantization. 
The algorithm is based on the utilization of GF ((7) -quantized 



LDGM codes, that permit to generate codewords with a 
non-uniform distribution. Eventually, this allows to directly 
compress sources with non-uniform distribution too. Among 
the appUcations, the proposed codes could be used in 
conjunction with good channel codes in problems of source 
coding with side information at the decoder 

The compression performances, obtained with regular de- 
gree distributions, show that the proposed algorithms perform 
reasonably close to the rate-distortion bound and always better 
than the time-sharing bound. It is also reasonable to assume 
that better performances could be achieved by suitably de- 
signing irregular degree-distribution codes, by augmenting the 
field cardinality q, or by employing damping strategies similar 
to the ones in fT2] for solving convergence problems, which 
are possible directions for future research on this subject. 
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